Since the inception of primitive information storage and communication through devices such as carvings in stone and wood, data has played an increasingly important role in society. In more modern times, inventions such as written language, the printing press, and photography have enabled communication between people on a global scale. Countless reams of history are stored on both printed and photographic paper. Since many of these paper documents are degrading rapidly, the risk of losing many primary source records documenting our modern history is considerable.
Damage to antique documents falls into the categories of mechanical and chemical. Mechanical damage includes defects such as cracks and scratches. Cracks generally do not have a common direction, whereas scratches usually have a specific orientation. Chemical damage occurs in the forms of semi-transparent blotches, gaps, and foxing. Semi-transparent blotches are frequently caused by water or humidity and cause each pixel to contain both noise and information about the original image. Gaps are caused by reactions between chemical agents and the film's gelatin, leaving a single gray level in a section of the image. Foxing creates red-brown areas due to microorganisms. Finally, the image may also be damaged when various materials are deposited on the picture or document's surface.
Many preservation efforts are been underway to protect antique images and documents. The first step has been to protect the original paper documents through means such as environmentally-controlled rooms. Furthermore, numerous documents are restored using chemical techniques. Generally, chemical methods start with a cleaning of the original, proceed to a physical restoration of the media, and conclude with some repainting of the restored media. Though this method improves image quality, it is clearly labor-intensive and the cost of restoring a single image is sizable.
As a solution to the cost problems associated with physical photograph restoration, digital restoration methods have been proposed. With the introduction of high-quality, low-cost color scanners and printers, restoring a photograph becomes a matter of scanning in the original, making modifications in the digital domain, and then either storing the image digitally or reprinting it. Though such an approach appears simple, modifying the scanned image in the digital domain is actually a complicated procedure. Currently, the most commonly used approach is to employ highly-trained operators and commercial software such as Adobe Photoshop or the GNU Image Manipulation Program to restore images. Though restoration through such software is less expensive than chemical restoration, one who is skilled in the art will readily appreciate that software restoration is a time-consuming process. Thus, beyond the cost of scanning the original, damaged documents, current digital photograph restoration still involves great operator expense.
Clearly a completely automated digital photograph restoration technique would reduce operator cost. Many automated and semi-automated approaches have been proposed and digital photograph restoration has been proven to be useful for the following reasons: restoration is reversible and the original document is not damaged; partial or complete automation reduces the amount of time for restoration, thus allowing for the processing of large quantities of images; and finally digital restoration is economical and affordable for any photographer or museum.
While a number of patents describing image restoration techniques have been issued, none provide an automated defect restoration system. U.S. Pat. No. 5,623,558 describes a system in which only completely undefined pixel locations are restored and where parts of the algorithm are not automated. U.S. Pat. No. 5,796,874 describes the restoration of faded images and requires user interaction. U.S. Pat. No. 6,487,321 describes a method for modifying defects in a digital image that are generally caused by analog-to-digital conversion and are not due to defects (e.g., blotches, cracks, and so on) in the original source image. U.S. Pat. No. 6,636,645 B1 describes a system in which noise and blocking artifacts are restored through the use of a user-specified noise table. U.S. Pat. No. 5,771,318 describes an edge-preserving smoothing filter, but does not restore old documents. U.S. Pat. No. 5,414,782 focuses on using partial restorations to retrieve details from images that are typically lost in other restoration techniques. Using these partial restorations, one can then tune parameters for other algorithms. However, this involves significant user interaction. U.S. Pat. No. 6,792,162 automatically enhances a digital negative, but cannot restore old documents. U.S. Pat. No. 6,879,735 presents a method for sharpening a blurred image, but again cannot be used for generalized old document restoration. U.S. patent application Ser. No. 10,887,998 removes semi-transparent artifacts from digital images, but only does so only when the artifacts are caused by contaminants in the optical path to the camera and is not capable of restoring old documents. U.S. patent application Ser. No. 11,236,805 describes as system of identifying and removing semi-transparent blotches; however, the system requires a clean reference image. The present invention does not require a clean reference image for the restoration of images damaged by semi-transparent blotches. U.S. patent application Ser. No. 12/032,670 describes an automated defect restoration system; however, the system only works for correcting dust marks that are generally due to a dusty camera lens and the system cannot restore semi-transparent blotches.
In addition to patents issued in the field of image processing, there is significant existing research on the subject of blotch removal. Existing work broadly falls into the categories of general photograph enhancement techniques, blotch removal from multi-frame film sequences, text enhancement, and blotch removal from single frames, as in the application of document restoration.
In general, several approaches to digital image restoration have been developed. One known approach referred to as histogram equalization, generates a restoration transfer function based upon the cumulative distribution function of the grey levels in an image. In order to overcome contrast losses associated with histogram equalization over the entire image, there is introduced a locally adaptive variant of histogram equalization. In another known approach, a more efficient algorithm can be used. Though this may address some of the performance issues with the approach of histogram equalization, it also demonstrates that histogram equalization may be useful for enhancing the overall contrast of an image and not for local defects such as semi-transparent blotches.
Adaptive nonlinear filters are also commonly used for image enhancement, which, as compared to prior works, improves contrast enhancement while performing noise reduction and edge enhancement. However, in the application of semi-transparent blotches, the parameters required for each blotch are different and unknown.
Another commonly used digital image enhancement technique is anisotropic diffusion. In another approach, a mathematical analysis of an anisotropic diffusion filter for ultrasound images is presented in which ultrasound noise is highly predictable and occurs in well-defined curves. This approach presents an adaptable ultrasound filter that significantly reduces noise in ultrasound images. Since the characteristics of the noise in semi-transparent blotches vary widely, these approaches are not suitable.
Adaptive contrast enhancement is another traditional image enhancement technique for the detection of blood vessels in retinal images. Adaptive contrast enhancement is used to only increase contrast in the area of blood vessels without creating background contrast objects. Though useful for the application of blood vessel detection, when applied to semi-transparent blotches, adaptive contrast enhancement frequently creates bright contrast objects that further corrupt the image.
Another approach to document and image defect removal is to replace damaged data with new pixels. Replacement approaches assume that the damaged areas of the image are completely lost and irrecoverable; therefore the only solution is to generate new data.
An example of a replacement approach includes using texture synthesis methods to generate new patterns to replace missing sections of data in an image. This does not require a regular prototype pattern, such as the mortar lines in a wall of bricks, and works well on natural images. However, this method is not effective for semi-transparent blotches.
Inpainting is another common replacement technique that may work well for repairing cracks in images. Further development of inpainting has been used to effectively remove limitations on the topology of the region to be inpainted. Yet another method of inpainting utilizes Gestaltist's Principle of Good Continuation to interpolate image data based on data from local gray levels and gradients. However, inpainting methods do not restore the original image data, which is a desirable feature for semi-transparent blotch removal.
In another approach, spatial and frequency domain information are utilized for noise reduction in images. Using a prototype image, there is replacement of the noise in an image with new pixels. Run iteratively, the algorithm can repair contiguous sections of the image effectively. However, this approach still replaces information, making it inappropriate for semi-transparent blotches.
In addition to the restoration of individual images, film restoration has seen considerable attention in research. Semi-transparent blotches are one type of defect in film and typically only affect individual frames. Most film restoration algorithms rely on information from several frames, thus making them unsuitable for the applications of document and photograph restoration.
For example, one approach detects blotches by assigning a probability to each pixel of being in a blotch. This probability is generated using both spatial and temporal analysis. Once the probabilities are assigned, they are used as a restoration parameter and anisotropic smoothing is performed to conclude the process. This approach is more robust to noise and errors than prior approaches; however, it does require training data, which is not available for semi-transparent blotches due to the inconsistent nature of each blotch.
Additionally, old film frequently has frame-alignment errors. Using a morphological detector, it can be assumed that the blotches are a local minima or maxima and removes the blotches using a motion estimator and multilevel median filtering. The algorithm performs blotch correction and contrast enhancement simultaneously. However, this requires information from multiple frames, which is not available for still photographs or documents.
In yet another approach, a system combines spatial and temporal information for blotch detection using Dempster-Shafer fusion. Using morphological functions to compare the detected shape with a prototype of a blotch, the system is able to effectively detect blotches in old film sequences, but not single-frame images.
In a further approach, rank ordered differences are employed to compare motion compensated frames and detect blotches and scratches in old film sequences. This approach provides better performance and lower computational load as compared to prior approaches. Similarly, another know system also detects blotches and scratches but utilizes heuristic and model-based methods. However, both methods require information from multiple frames.
In addition to a variety of image restoration techniques, many methods specific to text and handwriting have been developed: One such method introduces the Integral Ratio, which is a two-stage thresholding process designed to separate handwriting from various backgrounds. Another approach introduces a locally-adaptive, parameter-free binarization algorithm that extends upon Niblack's binarization using morphological and gradient-based error correction steps. However, text enhancement methods do not restore the background, thus making them unsuitable for semi-transparent blotch restoration. There has also been significant work in the area of text segmentation, which is a subset of the text binarization problem. One commercial application would be faxed documents, thus providing an incentive to study this problem. Though text binarization approaches effectively restore text, they do not consider the background. Background restoration is required for the removal of semi-transparent blotches, and thus text binarization approaches are not sufficient.
Of the known approaches that are specifically designed for the semi-transparent blotch removal, all have limitations. For example, some approaches leave visible borders around the perimeter of the blotch location and blotch detection involves considerable user interaction that improves upon detection but still requires user interaction for restoration. Another approach improves upon detection but is not user independent. A further approach improves upon other techniques, but assumes that the blotch is in a text document. Another approach presents a wavelet-based technique.